Levels of Representation

This article describes a class of theories, the distinguishing characteristic of which is that elements of each of a series of representational systems (levels) are constructed from those of an immediately subordinate, qualitatively distinct, system. Each of these ascending levels of representation (LOR) succeeds in modeling attributes of the environment that are not captured by lower levels (emergent properties); and each ascending level provides correspondingly novel mechanisms for the control of behavior. The class of LOR theories has instantiations as diverse as Aristotle's multilevel conception of the soul, Pavlov's two signal systems, and Piaget's genetic epistemology. This article describes applications to human cognition and behavior of a 5-level LOR theory that is based on recent cognitive research. Address correspondence to: Anthony G. Greenwald Department of Psychology, NI-25 University of Washington Seattle, WA 98195 Draft initially submitted to Psychological Review, April 27, 1987 Revision submitted to Psychological Review, May 16, 1988 Note: This draft contains several figures that will need copyright holders' permissions for reproduction if they are used in a published version. Some of these may be replaced by new, original figures. For comments on an earlier draft, the author is grateful to David A. Balota, Leonard Berkowitz, Jerome S. Bruner, Fergus I. M. Craik, Russell H. Fazio, L. Rowell Huesmann, Earl B. Hunt, Ulric Neisser, Thomas O. Nelson, John R. Palmer, Michael I. Posner, Anthony R. Pratkanis, David L. Ronis, Shelley E. Taylor, Endel Tulving, Gifford Weary, Delos D. Wickens, and several anonymous reviewers. Date of draft: May 13, 1988 (This copy includes minor corrections to submitted version) RUNNING HEAD: Levels of representation Greenwald: Levels of representation (Draft of May 13, 1988) -2LEVELS OF REPRESENTATION Section I of this article describes a general framework for theories of mental representation. Some of the many theoretical precursors of this levels-of-representation (LOR) approach are reviewed in Section II. Section III presents a 5-level LOR theory that is proposed as a model of human representational abilities, and Section IV describes several applications of this theory. Section V contrasts the class of LOR theories with alternative paradigmatic frameworks for representation theory and notes further possibilities for application of this class of theories. I. Representations and Levels In simplest terms, representation is a relationship of reference between two entities, which can be designated representation and referent. Often, but not necessarily, the representation is produced from the referent, as an audio recording is from an orchestral performance, or a library catalog card from a book. The representation is typically more accessible than the referent, and therefore can serve as a proxy for the referent. (Considering a card in a library's catalog and the corresponding book on its shelves, the card generally serves as proxy for the book, even though the reversal of these roles is conceivable.) The sense in which a representation serves as proxy for its referent varies considerably, as is suggested by the following examples: Referent Representation phoneme letter person name of person tables, chairs, etc. word: "furniture" scene photograph research observations theoretical statements Representational systems. Isolated referent-representation pairs are theoretically less interesting than are collections that involve similar reference relations. Some examples are the collection of cards that represent library books (a catalog), the collection of letters that represent phonemes (an alphabet), and the collection of words that represent objects, actions, attributes, etc. (a lexicon). In such systems of representations, each component representation has a relation not only to its referent, but also to other representation elements in the system. Because the relations among representations depend on characteristics of the medium in which they reside, a system of representations can have properties that its collection of referents lacks. Representational media. A system of representations occupies some substrate, or medium. Properties of the medium afford operations that can be performed on its resident representations. Two representational systems that share the same name can be functionally very different because they reside in different media, and therefore afford very different operations. For the example of a library's catalog, the operations that can be performed on catalog elements residing in a digital electronic medium are very different from those that can be performed on cards in file drawers. Similarly, a digitally-recorded symphony can be manipulated in ways that are quite impossible for the same symphony in printed score form. Operations. An operation can be described either as a process that converts one representation into another, or as an n-ary relationship that is specified as a set of n-tuples of elements, which can be understood as the operation's input(s) and output(s) (see Roberts, 1979). For example, the operation of adding 1 to a binary coded 4-bit integer can be described either (a) as a process, by starting at the rightmost (least significant) bit and changing ones to zeros until the first zero is encountered, which is changed to one, or (b) as the set of pairs {(0000,0001), (0001,0010), ..., (0111,1000), ..., (1110,1111)}. Between-medium operations transform a representation in one medium into a representation in another -such as the Greenwald: Levels of representation (Draft of May 13, 1988) -3The average telephone number in an office building is equally an emergent property of a numerical representation scheme, but it does not appear to be a useful one. There is a considerable body of theory, including some controversy, concerning the conditions under which statements about numerical representations are meaningful (see Michell, 1986). operation of printing xa photographic negative from film to paper. Within-medium operations produce output in the same medium as the input resides -for example, the adding-1 operation just described. Emergent properties. Medium-afforded operations endow representations with functional properties. For example, frames of cinematic film can be subjected to the operation of projection onto a screen in sequence, producing for the viewer the property of an illusion of movement. Properties that result from operations on representations are emergent in one or both of two senses: (a) they are not apparent in the latent (non-operated-upon) state of the representational unit, and (b) there may be no corresponding property in the referent domain. For the film example, motion of the image in ordinary playback is emergent in the first sense. The use of numbers to represent dimensional attributes -such as temperatures, lengths, or weights -permits emergence in the second sense, of properties that are associated with arithmetic operations on numbers. For example, the average age of a group of persons is a property that emerges only by virtue of a system for numerically representing a sequence of days. Levels of Representation The functions of a system of representations have been described as depending on two types of operations on the system's representational units: between-medium and within-medium operations. It is a third, and less familiar, type of operation with which this article is primarily concerned -combinatorial operations on representations, functioning to construct relations between systems of representations. This third type of relation/operation affords the possibility of an interrelated hierarchy of representational systems -in other words, the possibility of a system of levels of representation (an LOR system). The units of the lowest level of an LOR system derive from a between-medium operation that links the representational system to its referent domain. Units of each higher level are produced by composition operations for which units of the immediately lower level serve as input. Composition operations are asymmetric, resulting in a hierarchical ordering of the levels. Elements of each composed (higher) level are mapped onto multiple elements of the composing (lower) level, and are thus more complex than those of the lower level. Operations that apply to units of one level of an LOR system are (by definition of levels) different from those that apply to units of any other. (Units that are subject to the same operations are considered to be at the same level.) Accordingly, the composition operations that construct units of level n+1 from multiple units of level n are different from those that construct units of level n from multiple units of level n-1. As a consequence, units of level n+1 are qualitatively different from those of level n and have novel emergent properties. Three interrelated identifying characteristics of an LOR system are, then: (a) composition operations that combine multiple units of one level to form single units of the next higher level, (b) qualitatively distinct representational units at each level, and (c) novel emergent properties at each ascending level. A suitable format for describing an LOR system is one in which these three characteristics are indicated, as in Figure 1. Figure 1. A generalized levels-of-representation system having 3 levels. Greenwald: Levels of representation (Draft of May 13, 1988) -4Computer Processor/Memory Example Perhaps the best known example of an LOR system is one for which the medium is a computer. Figure 2 summarizes the major characteristics of such a system, designated LORc4 ("c" is for computer, and "4" for the system's four levels). The units at the four levels of LORc4 are bits, constants, variables, and functions. The operations that define units at each ascending level combine multiple units of the immediately lower level, in a fashion afforded by the computer's hardware. These operations are described here briefly and informally in terms of transformations that can be performed by a computer. Figure 2. Four levels of computer-system representation (LORc4). Constants are composed by combining n positionally distinct bits (where n is typically 8, 16, or 32). This composition (of constants) is a factorial combination of the n bits, the result of which is that 2n possible elements at the bit level (i.e., a 0 or a 1 at each of the n positions) produce 2 possible elements at the constant level. The composition operation for variables uses the computer's capability of shifting constants to and from memory registers (addresses) that are defined by a mapping of constants onto subregions of available storage space. A variable is therefore a relation of constants to a single memory address; the constants that are shifted in and out of that address are capable of playing the same role in computations. Although the number of elements at the variable level is potentially the same as that at the level of constants, the number of locations that serve as variables in most applications is typically much smaller than the number of constants. Functions are composed as relations among variables, which are implemented as sets of shifting and comparison processes on the contents of memory registers (i.e., on variables). For example, the division function is a relationship among variables that serve in the roles of dividend, divisor, and quotient. Note that functions can be described quite adequately as relations among constants, rather than as relations among variables (e.g., Roberts, 1979, p. 42). As an example, the square function for decimal integers can be described as the infinite set of pairs of constants {(1,1), (2,4), (3,9), (4,16), ...}. Such description of LORc4's 4th level (functions) in terms of relations among elements at its second level (constants) suggests that the 3rd level (variables) is unnecessary in this representational system. Although it is indeed not necessary, the level of variables is quite useful in describing the computer as a representational system. That is, functions are more aptly described as operating on the contents of specific memory registers, rather than as operations that work in some more direct sense on constants. Stated another way, describing the computer as a system that includes variables as a level intermediate between constants and functions can be useful in suggesting ways to construct such a machine. The possibility of omitting the variable level from the computer's description is equivalent to saying that the description of functions can be reduced from a description in terms of variables to one in terms of constants. Similarly, it is possible to reduce the description of functions to relations among positioned bits, undoing the LOR system entirely. These observations illustrate a general point: The number of levels in an LOR system is optional. This general point becomes significant in considering relations between multi-level systems and ones that use only a single form of representation, or none at all -these relations are discussed in the final section of this article. LORc4's lowest level is associated not only with input from a keyboard, but also with output to a second domain, a display screen. Consequently, results of the computer's operation can be communicated to an external world; this communication can be construed as behavior of LORc4. Such behavior of a representational system requires descending-direction operations that are indicated by arrows in Figure 2. Greenwald: Levels of representation (Draft of May 13, 1988) -5Some significant details. The computer-system example may seem inapt because the familiar structure of computer programs in high-level languages includes expressions that mix functions, variables, and constants. As an example, the BASIC-language expression (SQR(SQR(A)) * B) + 5 consists of one constant (5), two variables (A, B), and four functions (SQR [square root], SQR, * [multiplication], +). This expression appears to combine functions, variables, and constants as if they have interchangeable roles in computation. If functions, variables and constants do indeed play interchangeable roles in such expressions, then those three types of elements cannot be at different LOR levels, but must all be at the same LOR level (by virtue of the LOR definition that elements subject to the same operations are at the same level). Further examination makes clear, however, that expressions such as the one above do not consist of generally interchangeable elements. For example, although "B" in the above expression can be replaced by any constant, "A" can be replaced only by a nonnegative constant. Similarly, the inner square-root function can be replaced by a variable or a nonnegative constant, but the outer one cannot. A mixed-level formula such as the one above is in a state of partial evaluation; the evaluation process consists of replacing tokens of higher-level elements with ones for lowerlevel elements. It is not problematic for such a formula to contain tokens for elements at different levels when in a state of partial evaluation. The apparent recursiveness of the above expression may also seem to be problematic from the LOR perspective. In "SQR(SQR(A))" it appears that an element of LORc4's function level is composed as a relation involving another element of that same level (rather than of the level just below). Again, the LOR-legitimacy of the expression can be understood in terms of the process of evaluating such expressions. In evaluating "SQR(SQR(A))" the inner square-root must be replaced by a lower-level expression before the outer one is evaluated. This interpretation of an apparent (but not actual) recursive function uses a device parallel to one that Russell (1908) introduced into mathematical logic in order to resolve paradoxes of self-reference in set theory -his theory of types (i.e., levels). Visual System Example A 4-level visual LOR system (LORv4) has been described in Marr's (1982) computational analysis of vision. The four levels and their elements are illustrated in Figure 3. Each level uses a qualitatively different set of primitive units to represent a scene that is assumed to be sensed via an optical transduction (between-medium) operation such as video imaging. The medium in which LORv4 resides is one that affords computational operations corresponding to Euclidean distance relations (and differentiation of light intensity functions over such distances, etc.). Figure 3. Four levels of visual-system representation (from Marr, 1982). The levels (with constitutent primitive elements in parentheses) are: (a) image (pixels, or intensities at each point indicated by numbers in the 2-dimensional array of points), (b) primal sketch (showing a transition from clusters of pixels [top], to the addition of terminators [small filled circles], to oriented tokens, to groups of similar tokens [bottom], between which boundaries are constructed), (c) 21⁄2-D sketch (local surface orientations [short arrows], discontinuities in surface orientation [dotted lines] or depth [solid lines]), and distance from viewer [not indicated], and (d) 3-D model (axes to which generalized-cylinder volumetric primitives are attached). Greenwald: Levels of representation (Draft of May 13, 1988) -6Nonuniqueness of Descriptions of LOR Systems The description of an LOR system might include computational specification of composition and within-level operations that define emergent properties. The plausibility of such specification is indicated by the success of Marr's analysis, from which LORv4 was borrowed. However, computational descriptions of LOR operations are not unique. Rather, there is a mutual dependence between the choice of a structural description for representational units and the choice of a description for operations applied to those units (see Anderson, 1978, for a detailed example of this argument). For example, LORv4's lowest-level pixel units might be described equivalently as intensities at locations in a Cartesian coordinate system (projections on vertical and horizontal axes), or in a polar coordinate system (distance and direction from an origin), or in any of a potentially infinite number of other 2-spaces. Further, intensity values at these points can be described with various alternative numerical scaling methods. For each type of characterization of locations and intensities of pixels there will be a corresponding method of describing the composition operations that constitute blobs, edges, and other level-2 primitives, etc. Non-LOR Representational Hierarchies The concept of hierarchy in a representational system is well known in cognitive psychology. Some familiar representational hierarchies are shown in Figure 4. Importantly, none of the familiar hierarchies in Figure 4 is an LOR hierarchy. Rather, they are hierarchies in which the same composition relation joins each successive pair of levels. For each of the Figure 4 hierarchies, the several tiers collectively constitute just a single LOR level. The differences among the tiers of these hierarchies are differences in degree of representation of the same property. For example, in a taxonomic hierarchy such as that in Figure 4(a), higher tiers contain names of increasingly large subsets of the represented domain. Anywhere in the taxonomic hierarchy, a term designates a set that contains the sets (if any) linked to it in the downward direction and is contained within the set (if any) linked to it in the upward direction. Figure 4. Examples of non-LOR representational hierarchies for (a) taxonomic classsification (from Bower, 1970), (b) a fragment of a semantic network (from Collins & Quillian, 1969), and (c) detail in visual representations using generalized-cylinder components (from Marr, 1982). II. Levels of Representation in Psychological Theory The LOR conception appears at least partially in many psychological theories. Accordingly, a brief survey such as the one given here cannot attempt to be comprehensive. This survey is intended only to establish that recent theoretical developments in several areas of psychology are evolving toward explicit development of the LOR form; this article's description of LOR as a general theoretical framework continues and extends a long tradition in psychological theory. Philosophical roots. Perhaps the most distant identifiable precursor of the LOR framework is Aristotle's distinction (De Anima, Book II) among nutrition, appetite, sensation, locomotion, and thinking as a set of levels of functioning of the soul. Aristotle's levels of the soul were heirarchical in that (a) organisms did not necessarily possess all five of the functions, and (b) the presence of a given function implied the presence of the preceding ones. Some much more recent philosophical precursors are (a) the distinction, made by several associationist philosophers (notably Locke, Hume, and J. S. Mill) between impressions and ideas, the latter of which were more remote from experience, and (b) the related distinction between simple and complex impressions or ideas. In the "mental chemistry" of J. S. Mill (and, later, of Wundt and Greenwald: Levels of representation (Draft of May 13, 1988) -7other structuralist psychologists), it was explicitly claimed that complex impressions or ideas have novel (i.e., emergent) properties that exceed those of their more elementary constituents. Learning theorists. In the hands of Thorndike and Watson, behaviorism was, by design, a system that used a single level of explanation -that of associations between stimuli and responses. Pavlov (1955) applied the associative relation to a second level -his second signalling system. In the second-signal system, words (second-level conditioned stimuli) stand in the same relation to ordinary (first-level) conditioned stimuli as the latter do to unconditioned stimuli. Hebb's (1949) concepts of cell assembly and phase sequence similarly described two levels of "conceptual" nervous system organization in terms of structures at two levels of abstraction from experience. In the neobehaviorist theories of Hull (1943) and Spence (1956), fractional anticipatory responses -which, by virtue of their stimulus feedback, could represent imminent events -provided a means of specifying a second level of representation. Osgood (e.g., 1957) extended the fractional anticipatory response concept into the domain of word meaning. Each of the learning-theory systems just mentioned acknowledged two levels of representation of stimulus events. Although the operations or relations forming the second level were not always explicitly distinguished from those for the first level, novel emergent properties were generally credited to the higher level. One neobehaviorist system that made a point of identifying qualitatively distinct levels was Kendler's (1979) theoretical distinction between associations and hypotheses as "levels of functioning." Developmentalists. Karl Buhler (1930), noting that "Aristotle has anticipated us in the general idea of a sucession of levels in the psychic sphere" (p. 16), observed that On examining all ... purposeful modes of behaviour displayed by man and animals, we find a very simple and obvious structure consisting of three great stages in ascending order; these three stages are called instinct, training and intellect. (p. 2) Not only among developmentalists, but certainly also among all psychological theorists, Piaget (e.g., 1954; Inhelder & Piaget, 1956) has provided the most thoroughgoing conception of levels of representational activity. In Piaget's theory of cognitive development, the child is understood to progress from sensorimotor intelligence (first two years), through pre-operational thought (approximately ages 3-6), and concrete operations (7-12), to formal operations (over 13). The phenomena represented by the child not only increase in complexity through these four stages, but also the types of mental operations that are applied to the concepts change qualitatively and dramatically. A sign of the influence of Piaget's developmental theory is the number of subsequent writers who have suggested revisions (see, e.g., the review by Gelman & Baillargeon, 1983). Prominent among these, Bruner (1966) suggested a reformulation of the preoperational period (through age 6) into three (rather than two) stages, with successive development of enactive, iconic, and symbolic representation systems. Cognitive hierarches. The concept of hierarchy has been central to some of the major works of cognitive psychology (e.g., Miller, Galanter, & Pribram, 1960; Rosch, Mervis, Gray, Johnson, & Boyes-Braem, 1976; Selfridge, 1959; Simon, 1962). Accordingly, the principle that representations vary in their level of abstraction has become very familiar (see Figure 4). Until recently (see the second paragraph below), however, cognitive theorists made little use of the possibility of qualitatively different composition relations for units at successive stages of abstraction. For example, both Miller, 1956, and Simon, 1974, referred to the abstraction operation, regardless of its hierarchical level, by means of the single label, chunking. A major integrative application of the idea of hierarchical cognitive structure, merging contributions from perceptual and physiological psychology, appeared in Konorski's (1967) development of the concept of gnostic units. Information processing stages. The concept of an ordered series of stages of information processing was an early theoretical accomplishment of modern cognitive psychology, established by the work of Broadbent (1958), Sperling (1967), Sternberg (1967), Neisser (1967), Greenwald: Levels of representation (Draft of May 13, 1988) -8Norman (1968), and Smith (1968). The concept of stages was initially associated with a multistore conception of memory (sensory buffering, short term memory, long term memory), but has more recently developed into the conception of a succession of increasingly complex analytic operations applied to stimuli. This changing conception of stages can be seen, for example, in the evolution of the theory of levels of processing from Craik and Lockhart (1972), to Craik and Tulving (1975), to Craik (1983). A shared characteristic of several of the multistage theories is the distinction between a preattentive stage of processing, which occurs outside of consciousness and with (relatively) unlimited capacity, and attentional processing that is limited in capacity (see especially Neisser, 1967; also Treisman & Gelade, 1980, and Posner, 1985, for extensions of this distinction). Qualitative differences among representational systems. Explicit accounts of coexisting, qualitatively different representational systems have been offered in several theoretical treatments of human memory (e.g., Johnson, 1983; Sherry & Schacter, 1987; Tulving, 1983; see Tulving, 1985, for a review). Kosslyn's (1981) analysis of imagery appeals to three qualitatively different representational systems -literal, propositional, and imaginal. (See Anderson, 1978; Johnson-Laird, 1983; Palmer, 1978; Pylyshyn, 1973; and Shepard, 1978, for other analyses of representation that are designed to accommodate phenomena of mental imagery.) Several recent treatments have attempted to identify sets of elements (e.g., geons in Biederman, 1987; and features in Treisman & Gormican, 1988) that are subordinate to the visual perception of objects. The concepts of procedural and declarative knowledge as qualitatively distinct forms of representation has been developed both in research on artificial intelligence (e.g., Anderson, 1983) and in analyses of the limited learning capabilities of various types of amnesics, particularly Korsakoff-syndrome patients (e.g., Graf, Squire, & Mandler, 1984; Jacoby & Witherspoon, 1982). Tulving (1985) has integrated a number of these developments into a theory in which memory is described as a tripartite system that is hierarchical in form: The system at the lowest level of the hierarchy, procedural memory, contains semantic memory as its single specialized subsystem, and semantic memory, in turn, contains episodic memory as its single specialized subsystem. In this scheme, each higher system depends on, and is supported by, the lower system or systems, but it possesses unique capabilities not possessed by the lower systems. (p. 387, emphasis added) This brief survey establishes that a great variety of psychological theories appeal to hierarchical systems of representational levels (some other related theories are considered in the final section of this article). In many of these theories, the representational units at different levels are qualitatively distinct, as in an LOR system. However, there is little consensus among theorists regarding the number or identity of levels. III. A Five-Level System of Psychological Representations Figure 5 summarizes a 5-level system (LORh5 -"h" for human) that is discussed in the remainder of this article. The five levels of LORh5 are features, objects, categories, propositions, and schemata. A brief and somewhat intuitive description of each level's composition operations and emergent properties is followed by more careful descriptions of the five levels. The first level, features, consists of primitive sensory qualities such as pitch, hue, brightness, loudness, odor, and taste, along with properties that constitute the building blocks of visual, tactile, and auditory pattern (e.g., linearity, angularity, curvature, sharpness, voicedness, nasality). Features constitute LORh5's lowest level, and are not compounds of more primitive units; they are the elementary properties that are sensed by LORh5. Greenwald: Levels of representation (Draft of May 13, 1988) -9Figure 5. Five levels of human mental representation (LORh5). Objects in visual space are composed of features that move together in rigid groupings relative to a background. The composition operations for objects can be described intuitively by gestalt grouping relations such as proximity, similarity, and common fate of features (Koffka, 1935). The chief emergent property of the object level is physical identity, the conservation of object-ness through physical transformations of features such as rotation, occlusion, and change of illumination. Object representations are combined by the relation of class membership into categories. Establishment of categories must be aided by adults' behavior, in the presence of children, of pointing to and providing (category) names for objects, as well as by the process of contextual generalization that was described by Braine (1963). The major emergent property of the system of category representations is abstract identity, the relation between objects that share category membership. An emergent property that depends jointly on category and object representation is number, which is a property of collections of objects that are identical at the category level (e.g., apples) but distinct at the object level. Propositions combine a function (e.g., action) category together with a suitable set of argument (e.g., agent, object, instrument, attribute) categories. It is plausible that proposition representations develop in association with adults' use, in the presence of a child, of word strings to accompany their own, others', or the child's behavior. Emergent properties of the system of propositional representations are denoted by terms that describe relations among the categories that consitute a proposition -for example, agency (the relation between actor and action) and instrumentality (the relation between object and action). There are several familiar types of relationships among propositions that define schemata. Script schemata (Schank & Abelson, 1977) consist of propositions in sequences that are appropriate for narrative descriptions; frame (Minsky, 1975) or model (Johnson-Laird, 1983) schemata are coherent sets of propositions that describe attributes of an object or situation; causal schemata consist of propositions in a plausible causal ordering; and logical schemata are propositions in sequences that adhere to the formal rules of an axiomatic system. Consistency is a general designation for the class of emergent properties of schema representation; propositions that are eligible to co-participate in a schema are mutually consistent. Among the great variety of forms of consistency for different types of schemata are narrative coherence, analogy, logical proof, cognitive consonance (Festinger, 1957), cognitive balance, (Heider, 1958), self-consistency (Lecky, 1945), legality (consistency with a body of laws or rules), and empirical validity (consistency with a body of data). Similarity and Levels of Representation By virtue of their distinctive representational units and relations, each level of LORh5 supports a different basis for identifying equivalence and similarity between events. For example, changing the color of light in which a white chair is seen creates a difference at the feature level, but not at the object level. Two wooden chairs are different at the object level but equivalent at the category level. Etc. By well-established convention, similarity judgments are understood to reveal mental structure (e.g., Shepard, 1980; Tversky, 1977). Accordingly, conclusions about mental structure based on similarity judgments should depend importantly on the level of representation that research participants, by virtue of ability, materials, or instructions, apply to the analysis of stimuli presented for similarity judgments. At the feature level, the basis for similarity is stimulus generalization, which can be understood as a measure of the overlap of features activated by two events, expressible as distance in multidimensional space (see Shepard, 1987). At the level of objects, similarity in the form of object identity occurs when spatiotemporally separated feature collections activate the Greenwald: Levels of representation (Draft of May 13, 1988) -10same object representation. Among category representations equivalence occurs in the form of abstract identity, by which physically nonidentical objects are treated as equivalent. Scenes or events may be said to have functional identity when they share a propositional representation. Formal identity occurs when different sets of propositions are schematically equivalent, as when two logical arguments use the same form of syllogism. A puzzling task-dependent reversal in the asymmetry of similarity judgments may be explained by appealing to task differences in level of representations. In the task of searching for a figure in a uniform background of different figures (Treisman & Gormican, 1988), deviantfrom-prototype figures "pop out" from backgrounds textured by a prototypical figure, but not vice versa; in this task, the deviant element (e.g., an ellipse) appears more different (pops out) from a prototypical element (e.g., a circle) than vice versa. However, in verbal judgment tasks (e.g., Rosch, 1975; Tversky, 1977) exactly the reverse pattern is found, with the deviant element being judged more similar to the prototypical one than vice versa. This paradoxical difference between tasks may be explained by assuming that the visual search and verbal similarityjudgment tasks employ qualitatively different systems of representations (presumably, feature and category levels, respectively). Specification of LORh5's Interlevel Relations Relations among the elements of a given LOR level constitute both the representational properties of that level and the elements of the next higher level. Because the domain of LORh5 encompasses virtually all of psychology, it is not surprising that each of LORh5's four interlevel relations corresponds to an extensive body of existing research. The present descriptions of interlevel relations proceed by considering various existing approaches to these interlevel relations, and attempting to extract their shared structural properties. Because of the magnitude of relevant prior work, citations of previous works that have influenced this analysis can only be superficial. Figure 6. Structural models of the composition of objects from features. (a) Selfridge's (1959) Pandemonium model (features = "data demons"; objects = "cognitive demons"), (b) Uhr and Vossler's (1963) pattern-recognition program showing an operator (feature) being applied to a to-be-classified image (the operator is matched to all possible locations on the image at left; two matches are described in terms of measures of their horizontal position [X], vertical position [Y], and squared radial distance from the center of the image [R]), (c) Jakobson and Halle's (1956) construction of English-language phonemes from distinctive features (from Gibson, 1969), and (d) Rumelhart, Hinton, and Williams's (1986) connectionist multilayer network for pattern analysis ("input patterns" = features; "output patterns" = objects). Features to objects. Identities of elementary features hypothesized to be involved in construction of objects have been described for printed letters (Gibson, 1969, Neisser, 1967), speech (Jakobson & Halle, 1956), solid objects (Biederman, 1987; Julesz & Bergen, 1983; Marr, 1982), moving rigid objects (Shepard, 1979), and moving organisms (Johansson, 1973). Theorizing about the form of composition relations for object perception has been given (to mention just a few) by Garner (1974), Selfridge (1959), Morton (1969), Uhr and Vossler (1963), and Shepard (1987), and is proliferating in recent work on parallel distributed processing (connectionism; see Rumelhart, McClelland and the PDP Research Group, 1986). The common structural property in theoretical interpretations of the composition of features into objects is an associative network, some portrayals of which are given in Figure 6. A single-layer Greenwald: Levels of representation (Draft of May 13, 1988) -11network is easily described as a 2-dimensional matrix, in which rows represent input features and columns represent objects. Although this matrix description is easily generalized to multiple layers (with outputs of one layer becoming inputs of the next), multilayer matrix structures are not as clearly displayable as are formally equivalent node and arc structures (two examples of which are shown in Figure 6). Objects to categories. In various theoretical treatments, categories are assumed to be constructed as (a) points or regions in a multidimensional feature space, (b) lists of propositions, or (c) collections of objects (see the review by Smith & Medin, 1981). The multi-feature interpretation of categories uses the structure just described for feature-to-object relations, and the proposition-list interpretation uses a structure that is described below for proposition-toschema relations. The interpretation of categories as collections of objects (the "multiexemplar" view as identified by Smith & Medin, 1981) is the one adopted here for LORh5. Some portrayals based on the collection-of-objects interpretation are shown in Figure 7. The critical property of Figure 7's representation is an n:1 (members:category) structure. This n:1 tree structure contrasts with the n:m (features:objects) associative network structure of Figure 6. One consequence of the structural difference is that a 1-layer, n-element collection of (binary) features can encode a maximum of 2 objects, whereas a 1-layer, n-element collection of objects can encode, at the maximum, only a somewhat smaller nimner (2-n-1) of categories (assuming a minimum category size of 2 objects). Although these theoretical limits suggest that object:feature ratios are only slightly greater than category:object ratios, in practice the ratios are vastly different. For example, spoken English can be analyzed into a 2-layer feature-toobject structure, with fewer than 10 articulatory distinctive features combining into about 30 phonemes (see Figure 6), and these combining into vocabularies of more than 10,000 words for adult speakers, indicating an object:feature ratio in excess of 1000:1. In contrast, it is difficult to think of more than a few categorizations for the objects in familiar medium-sized collections (such as the letters of the English alphabet, the musical instruments in an orchestra, or the objects in a living room), suggesting that typical category:object ratios are lower than 1:1. Figure 7. Structural models of the composition of categories from objects. (a) Rosch's (1978) hierarchy of subordinate, basic-level, and superordinate categories, (b) Keil's (1979) ontological tree, and (c) Sattath and Tversky's (1977) empirically derived tree structure for mammals, based on similarity data from Henley (1969). Categories to propositions. Importantly influenced by Fillmore's (1968) analysis of grammar, recent work in cognitive psychology and psycholinguistics has increasingly conceived the composition of propositions in terms of verb-centered case structures, rather than the previously favored subject-predicate structures. Although notations vary considerably, the verb-centered structure -which might also be described as a function-plus-arguments structure -is discernible in the structural formats for propositions that are illustrated in Figure 8. Propositions to schemata. Schematic representation is assumed to support the broad range of accomplishments that are characterized as human rationality. Although there are many existing analyses of processes that are schematic (in the LORh5 sense), in relatively few of these have there been clear distinctions between propositional and schematic relations (an exception is Schank and Abelson's, 1977, distinction between conceptualizations [propositions] and conceptual dependencies [schemata]). The relations that define schemata can be formulated as sets of rules (i.e., of propositions) that describe the dependency of one proposition's content on the content of one or more other propositions. Some examples of schema-forming rules are given in Figure 9. Greenwald: Levels of representation (Draft of May 13, 1988) -12Figure 8. Structural models of the composition of propositions from categories. (a) Schank and Abelson's (1977) description of active and stative conceptualization (proposition) forms, (b) Norman, Rumelhart, and the LNR Research Group's (1975) notation for n-ary relations, illustrated with the sentence, "Mary told Helen that she gave John a dollar" (the give proposition is the object of the tell proposition), (c) Anderson's (1980) 2-proposition representation of "The early bird catches the worm" (the oval symbols represent the proposition units), and (d) Kintsch's (1974) 8-proposition representation of the sentence-text, "Cleopatra's downfall lay in her foolish trust in the political figures of the Roman world." Figure 9. Structural models of the composition of schemata from propositions. (a) notations for causal links between propositions and an example of their use ("rE" indicates the "r" and "E" relations in succession, with the intermediate state not shown), from Schank and Abelson (1977), (b) the balance schema for relations among three propositions, each denoting a liking (+) or disliking (-) relation (represented as a directed link) from a person (Perceiver or Other) to an object (O or X) (the set of three relations is consistent with the balance-theory schema when there are either 0 or 2 disliking relations), based on Heider (1946), and (c) schemata for the valid syllogisms of traditional logic (A, E, I, and O designate forms of propositions, e.g., "A" is a uniform affirmative such as "Every X is a Y"; the four "figures" are forms of distributing the major term, minor term, and middle term among the roles of subject and predicate in the two premises; and the "names" are mnemonics that encode the order of proposition forms in each syllogism) from Brody (1967); see also Prior (1967). Operations and Emergent Properties The representational structures at each level afford operations that properly analyze partial or noisy information. These operations are of the sort that Bruner (1957a; 1957b) identified as "perceptual readiness" and "going beyond the information given." A greater theoretical challenge is to describe operations that explain each level's emergent properties. Object level. The structure that supports object representation must explain various imaginal abilities, such as mental rotation and melody transposition, and the ability to conserve object identity through changes in perspective and partial or complete occlusion. These feats appear to require structures that can extract dimensional information and compute transformations along these dimensions. Suitable representational machinery is found in work on scaling (see Shepard, 1980, for an overview). But it may require substantial theoretical work to integrate the feature-based network structure (cf. Figure 6) with the capability for geometric transformations. Category level. The basic operation afforded by category representation is abstraction (finding a category to which an object belongs). Other operations that provide for emergent properties of the structure are (a) instantiation (finding a member of a category), (b) finding a comember (abstraction followed by instantiation), and (c) location of an overlapping category (instantiation followed by abstraction). Proposition level. The most familiar operations that can be applied to propositions are grammatical transformations of tense, mood, and voice. These emergent properties, and the structures required to support them, may be partly appreciated by analogy to familiar operations on function-plus-argument structures in mathematics -the integration and differentiation transformations of the calculus. It is plausible that cognition of temporal relations depends on Greenwald: Levels of representation (Draft of May 13, 1988) -13propositional structures. It may be apparent that existing structural portrayals of propositions (see Figure 8) provide little suggestion as to how the emergent properties of propositional representation, such as agency, instrumentality, and temporality, are computed. Schema level. The use of schema structures to produce propositions, as in the derivation of theorems from axioms or conclusions from premises, corresponds to one of the uses of productions in some treatments of artificial intelligence (e.g., Anderson, 1982; Newell, 1973). The schema structure also must compute whether or not a configuration of propositions is consistent with the schema rule. The representational constructions that philosophers of mind refer to as propositional attitudes (e.g., the belief in the truth of a proposition, or the desire that a proposition be true) may also be conceived as products of schema-level operations. Schemalevel processes are also importantly involved in ordinary comprehension of language; the meaning of a sentence may often depend more on its role in a familiar schema structure than on its proposition-level interpretation. (See Figure 10, which reinterprets a proposition from Figure 8 by assimilating it to a schema structure.) Figure 10. An illustration of LORh5's levels applied to interpretation of The early bird catches the worm. The components of this sentence can be related to representational entities at lower (feature, object, and category) levels. However, the commonly understood meaning is a causal schema consisting of four propositions. In abstract form, the four propositions (and their causal links) are: Agent A performs some action (which results in) A is early (which enables) A performs some [second] action (which results in) A attains some desired goal. By virtue of this schema structure, the meaning is more similar to A stitch in time saves nine than to (say) The best time for worm-hunting is in the morning. Some Comments on LORh5 Incompleteness of the description. Full description of a set of levels of representation should include detailed computational specification of the operations or relations that compose and transform the representational units at each level. The possibility that a system like LORh5 can be computationally specified, even though the effort of doing so is certainly massive, is suggested by the success of computer programs that model substantial portions of multi-level representational systems (e.g., Marr, 1982; Schank & Abelson, 1977; Winograd, 1972). Relation to LORc4. It is not accidental that the first four levels of LORh5 closely parallel the four levels of LORc4. The parallel may be extended by supposing a fifth level added to LORc4, to accommodate such computer capabilities as error-identifying program compilation and comprehension of natural language. Qualitative distinctness of levels. In measurement theory, qualitative differences in representations (for example, the differences among nominal, ordinal, and interval scales) result from differences in the relational structures that underly these representations (Krantz et al., 1971; Roberts, 1979). Similarly, qualitative distinctness of LOR levels depends on differences in underlying relational structures. LORh5's relational structures, which are approximated by the diagrams in Figures 6 through 9, are more complex than those that underlie measurement representations. Correspondingly, the qualitative distinctness of LORh5's levels can not yet be characterized other than in the informal terms used above in describing levels and operations. The qualitative distinctness of LORh5's levels provides a novel perspective on a type of definitional problem in which cognitive theorists occasionally become entangled. The problem is that of attempting to define a representational concept at one level in terms of relations that are more appropriate for some other level. For example, a few decades ago much effort was Greenwald: Levels of representation (Draft of May 13, 1988) -14spent on producing sets of grammatical rules that could generate or (in reverse) parse all utterances of a natural language. In the LORh5 framework, this effort is prejudged as inappropriate, because it attempts to use schema-level relations (e.g., an axiom-based set of grammatical rules) to describe units of another (proposition) level that are not expected to be so describable. Similarly, attempts to define categories in terms of rule-specified criteria for membership, or in terms of the multidimensional locus of feature combinations, are inappropriate in the LOR context (cf. Smith & Medin, 1981, for a review of the conceptual and empirical difficulties of such category definitions). Possibility of additional levels. One can conceive of additional levels within the span of LORh5's five levels. For example, LORv4 can plausibly amplify the transition between LORh5's adjacent feature and object levels. Likewise, the levels of LORv4 could be further elaborated by embedding within it levels such as those described by Hubel and Wiesel (1963). Abelson (1973) suggested a set of six levels that span the present bridge between proposition and schema levels. Consequently, it appears that the number of levels used in a theoretical analysis of representation is optional, constrained chiefly by one's taste for detail. Testability of LOR formulations. The LOR conception is a theoretical schema -a set of rules for describing theories of mental representation rather than a specific, testable theory. Any specific version of the LOR schema (such as LORc4, LORv4, or LORh5) is testable, in the sense that, when accompanied by operational definitions of units and relations, it can be examined for agreement with empirical observations. The next section provides further specification of LORh5, increasing its testability as a theory of mental representation. However, this article aims less to assert the empirical validity of LORh5 than to establish the usefulness of the LOR framework as a theoretical schema for psychology. IV. Applications of the LOR Framework How is a theory in the LOR form potentially more useful than one in nonhierarchical form? What justification is there for the specific identities of the five levels in LORh5? The range of the following applications indicates both that (a) a theory in the hierarchical LOR form has the potential to offer a unified theoretical treatment of phenomena that are not clearly related in other representation-theory formats, and (b) the five levels of LORh5 can be associated with empirical phenomena in a broad range of domains.